DEV Community: David Li

Searching for news with google news rss feeds

David Li — Thu, 07 Dec 2023 00:00:00 +0000

Google News is a news aggregator and personalized news service developed by Google. It was launched in 2002 and has since become one of the largest news aggregators on the web. The service collects news articles from thousands of sources, including national and international news organizations, as well as local and regional publishers, and presents them in a format that makes it easy to quickly find and read the news that interests you. Google News uses algorithms to analyze and rank news articles, ensuring that the most relevant and up-to-date articles are displayed first. The service is free and can be accessed through a web browser or mobile app.

You can use the Google News RSS (Really Simple Syndication) feed to find news articles based on a custom search query. The basic format for using the Google News RSS feed is as follows:

https://news.google.com/rss/search?q=SEARCH_QUERY

Replace "SEARCH_QUERY" with the keyword or phrase you want to search for, and include it in the URL. For example, if you want to find news articles about technology, you can use the following URL:

https://news.google.com/rss/search?q=technology

You can use this URL to subscribe to the RSS feed using a news reader or aggregator, or you can access the feed directly in your web browser to see a list of news articles related to your search query.

Note that the Google News RSS feed is subject to change and may not be available in all countries. Also, be aware that the terms of service for Google News state that you may not use the service to scrape or display its content on another website or application without permission.

For the full source code please view

https://github.com/FriendlyUser/news-alert/blob/master/News.cs

 using System;
//Request library
using System.Net;
using System.IO;
using System.Web;
using System.Xml;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Collections.Generic;
using System.Threading.Tasks;
using Elasticsearch.Net;
using System.Net.Http;
namespace news_alert
{
    public class News
    {
        public static readonly HttpClient client = new HttpClient();

        static public void SendItemToDiscord(XmlNode item) {
            // set discord webhook as private env
            string webhook = Environment.GetEnvironmentVariable("DISCORD_WEBHOOK");
            string esInstance = Environment.GetEnvironmentVariable("ES_INSTANCE");
            if (webhook == null) {
                Console.WriteLine("GET A DISCORD WEBHOOK");
                return;
            }
            var request = (HttpWebRequest)WebRequest.Create(webhook);
            request.ContentType = "application/json";
            request.Method = "POST";
            DateTime pubDate = Convert.ToDateTime(item["pubDate"].InnerText);
            string postLink = item["link"].InnerText;
            string discordTemplate = "{0} \n {1} \n {2} ";
            string discordMessage = string.Format(discordTemplate, item["title"].InnerText,
                item["pubDate"].InnerText, postLink);
            // write message of data sent to discord
            var w = new WebhookData() { content = discordMessage };
            using (var streamWriter = new StreamWriter(request.GetRequestStream()))
            {
                streamWriter.Write(JsonSerializer.Serialize<WebhookData>(w));
            }
            var httpResponse = (HttpWebResponse)request.GetResponse();
            using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
            {
                var result = streamReader.ReadToEnd();
            }
            // send data to es instance
            var settings = new ConnectionConfiguration(new Uri(esInstance))
                .RequestTimeout(TimeSpan.FromMinutes(2));
            var lowlevelClient = new ElasticLowLevelClient(settings);
            var post = new Post
            {
                guid = item["guid"].InnerText
            };
            var asyncIndexResponse = lowlevelClient.Index<StringResponse>("post", PostData.Serializable(post)); 
            string responseString = asyncIndexResponse.Body;
            Console.WriteLine(responseString);
        }

        static async Task Main(string[] args)
        {
            await checkNews();
            await Task.Run(() => checkPrices());
        }
        static async public Task checkPrices() {
            string stockUrl = Environment.GetEnvironmentVariable("STOCK_URL");
            if (stockUrl == null) {
                Console.WriteLine("GET A Stock URL WEBHOOK");
                return;
            }
            List<StockData> stocks = new List<StockData>();
            stocks.Add(new StockData() {ticker="NEXCF", targetPrice=1.60});
            foreach (StockData stock in stocks)
            {
                Console.WriteLine(stock.ticker);
                // query api
            }
            // either return a completed Task, or await for it (there is a difference!
            await Task.CompletedTask;
        }
        static async public Task checkNews() {
            var searchItems = new List<string> { "hydrogen",
                "analyticGPT", "natural gas", "recession",
                "Tech layoffs", "Bard AI"
            };
            foreach (string searchText in searchItems)
            {
                // line of items of cli list
                string xml = await FetchData(searchText);
                XmlDocument doc = new XmlDocument();
                doc.LoadXml(xml);
                // only print the first few elements
                XmlNodeList items = doc.GetElementsByTagName("item");
                DateTime utcDate = DateTime.UtcNow;
                for (int i=0; i < 5; i++)
                {   string pubDateStr = "";
                    try {
                        pubDateStr = items[i]["pubDate"].InnerText;
                    } catch (NullReferenceException e) {
                        Console.WriteLine("\nException Caught!");   
                        Console.WriteLine("Message :{0} ",e.Message);
                        continue;
                    }
                    DateTime pubDate = Convert.ToDateTime(pubDateStr);
                    // Console.WriteLine(pubDate);
                    if (utcDate.Subtract(pubDate).TotalHours < 24 * 2) {
                        Console.WriteLine("Within 2 days");
                        // check if id is in db
                        string esInstance = Environment.GetEnvironmentVariable("ES_INSTANCE");
                        var settings = new ConnectionConfiguration(new Uri(esInstance))
                            .RequestTimeout(TimeSpan.FromMinutes(2));
                        var lowlevelClient = new ElasticLowLevelClient(settings);
                        var searchResponse = lowlevelClient.Search<StringResponse>("post", PostData.Serializable(new
                        {
                            query = new
                            {
                                match = new
                                {
                                    guid = items[i]["guid"].InnerText// items[i]["guid"].InnerText
                                }
                            }
                        }));
                        var successful = searchResponse.Success;
                        var responseJson = searchResponse.Body;
                        SearchResult searchResult = JsonSerializer.Deserialize<SearchResult>(responseJson);
                        if (searchResult.hits.total.value == 0) {
                            SendItemToDiscord(items[i]);
                        } else {
                            Console.WriteLine("Match Already in DB, not going to print");
                        }
                    }
                }
            }
            await Task.CompletedTask;
        }
        static async public Task<string> FetchData(string searchText) {
            var query = HttpUtility.ParseQueryString(string.Empty);
            query["q"] = searchText;
            string queryString = query.ToString();

            string htmlData = string.Empty;
            string urlTemplate = @"https://news.google.com/rss/search?{0}";
            string url = string.Format(urlTemplate, queryString);
            try 
            {
              string responseBody = await client.GetStringAsync(url);

              Console.WriteLine(responseBody);
              return await Task.FromResult(responseBody);
            }
            catch(HttpRequestException e)
            {
              Console.WriteLine("\nException Caught!"); 
              Console.WriteLine("Message :{0} ",e.Message);
            }
            return await Task.FromResult("");
        }
    }
}

This code is a .NET Core console application that performs two main functions: checking for news articles and checking for stock prices. The news checking function fetches the latest news articles from an API and sends a summary of the latest articles that contain keywords from a predefined list of search terms to a Discord channel. The function also sends the information to an Elasticsearch instance for tracking. The stock price checking function queries a stock API to get the latest stock prices of a predefined list of stocks and logs the results.

Here's a high-level overview of the code:

The "News" class contains the main logic for the application.
The "SendItemToDiscord" method sends a summary of a news article to a Discord channel. The method uses the Discord webhook URL, which is stored as an environment variable, to post a message to a specific Discord channel. The method also uses the Elasticsearch instance URL, stored as another environment variable, to index the article for tracking purposes.
The "checkNews" method checks for the latest news articles that contain the keywords from a predefined list of search terms. The method calls the "FetchData" method to retrieve the latest news articles in XML format and parses the XML to extract the article summaries. The method then sends the latest articles to the Discord channel using the "SendItemToDiscord" method.
The "checkPrices" method checks for the latest stock prices of a predefined list of stocks. The method uses the stock API URL, stored as an environment variable, to retrieve the latest stock prices. The method logs the results of the stock prices.
The "Main" method is the entry point of the application and calls the "checkNews" and "checkPrices" methods. The "checkNews" and "checkPrices" methods are executed as asynchronous tasks.

Using RSS feeds as a news source for investing purposes can be a useful way to stay up to date on the latest financial news and market developments. This can be especially useful for investors who are interested in a particular sector or industry, as they can subscribe to RSS feeds from news sources that specialize in that area.

By subscribing to these feeds, investors can receive notifications as soon as new articles or reports are published, which can provide them with valuable insights and information about the market and specific investments. Additionally, investors can use RSS feeds to track the latest news on companies they are interested in, which can help them stay informed about important developments and trends.

However, it is important for investors to keep in mind that not all news sources are created equal. Some sources may have biases or agendas, and it is important to be critical and evaluate the information being presented. It is also important to seek out multiple sources of information, as relying solely on one source may lead to a skewed or incomplete view of the market and individual investments.

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. It is used for various purposes, including:

Full-Text Search: Elasticsearch is particularly useful for full-text search queries, with its ability to search through large volumes of data quickly and efficiently.
Analytics and Business Intelligence: Elasticsearch can be used to gather insights from data and to perform advanced analytics tasks, such as aggregations and data analysis.
Log Management and Analysis: Elasticsearch is widely used for log management and analysis, making it possible to quickly search through large amounts of log data to identify patterns and troubleshoot issues.
Application Performance Monitoring: Elasticsearch can be used to monitor the performance of applications, as well as to store and analyze log data, providing real-time visibility into the health of the application and its components.
Geospatial Data: Elasticsearch has built-in support for geospatial data, making it possible to search, filter and aggregate data based on location.
E-commerce: Elasticsearch can be used to power search and recommendations in e-commerce applications, helping users find the products they are looking for quickly and easily.
Security Information and Event Management (SIEM): Elasticsearch can be used as part of a SIEM solution to collect, store, and analyze security-related data from various sources, such as network devices and applications.

These are just a few of the many use cases for Elasticsearch, and it is a highly versatile technology that can be used in a wide range of applications and industries.

Yes, Bonsai.io is one of the providers that offer free hosting for Elasticsearch. With their free plan, you can host a single cluster with up to 3GB of storage and 20GB of bandwidth per month. However, it's worth noting that the free plan is intended for experimentation and learning, and may not be suitable for production-level workloads. Bonsai.io also offers paid plans with more advanced features, higher storage, and better performance.

Alfred backend with openfoodfacts

David Li — Thu, 30 Nov 2023 00:00:00 +0000

This post is related to my last post at https://friendlyuser.github.io/posts/tech/flutter/how_to_deploy_dart_to_hs.html. I will be using the same backend to create a wrapper around the openfoodfacts api. The openfoodfacts api is a free and open source api that provides information about food products. The api is available at https://world.openfoodfacts.org/data. The api is available in different languages and I will be using dart.
Food facts are important for a variety of reasons. First and foremost, they help us understand what we are eating and what nutrients our body is receiving. This information can be used to make informed decisions about our diets, which can impact our health and well-being.

Food facts also play a role in food labeling, which is regulated by government agencies to ensure that consumers have accurate information about the food they are purchasing. This can include information about ingredients, allergens, and calorie content. This helps consumers with specific dietary needs, such as food allergies or weight management, make informed choices.

In addition, food facts can be used to promote public health. For example, information about the sodium content of processed foods can be used to raise awareness about the risks of high sodium diets, and promote the consumption of healthier alternatives.

Overall, food facts are an important tool for empowering consumers to make informed decisions about their diets and for promoting public health.

import 'package:openfoodfacts/openfoodfacts.dart';

Future<SearchResult> search(Map<String, String> query) {
  var parametersList = <Parameter>[];
  // check for terms in query

  if (query.containsKey('terms')) {
    var terms = query['terms'];
    if (terms != null && terms.isNotEmpty) {
      parametersList.add(SearchTerms(terms: terms.split(',')));
    }
  }
  // check for withoutAddictives
  if (query.containsKey('withoutAdditives')) {
    var withoutAdditives = query['withoutAdditives'];
    if (withoutAdditives != null && withoutAdditives == "true") {
      parametersList.add(WithoutAdditives());
    }
  }

  if (query.containsKey('sort')) {
    var sort = query['sort'];
    if (sort != null && sort.isNotEmpty) {
      var option = SortOption.values.firstWhere(
          (e) => e.toString() == sort,
          orElse: () => SortOption.PRODUCT_NAME);
      parametersList.add(SortBy(option: option));
    }
  }

  if (query.containsKey('pnnsGroup2')) {
    // print('pnnsGroup2: ${query['pnnsGroup2']}');
    var pnnsGroup2 = query['pnnsGroup2'];
    if (pnnsGroup2 != null && pnnsGroup2.isNotEmpty) {
       var option = PnnsGroup2.values.firstWhere(
          (e) => e.toString() == pnnsGroup2,
          orElse: () => PnnsGroup2.PIZZA_PIES_AND_QUICHE);
      parametersList.add(PnnsGroup2Filter(pnnsGroup2: option));
    }
  }

  // page size
  if (query.containsKey('size')) {
    var size = query['size'];
    if (size != null && size.isNotEmpty) {
      parametersList.add(PageSize(size: int.parse(size)));
    }
  }

  // page number
  if (query.containsKey('page')) {
    var page = query['page'];
    if (page != null && page.isNotEmpty) {
      parametersList.add(PageNumber(page: int.parse(page)));
    }
  }

  // by tags
  if (query.containsKey('tags')) {
    var rawTags = query['tags'];
    if (rawTags != null && rawTags.isNotEmpty) {
      var tags = rawTags.split(',');
      // iterate over tags
      for (var tag in tags) {
        // check for tag with value
        // check tag
        if (tag.contains(':')) {
          var tagParts = tag.split(':');
          var tagName = tagParts[0];
          var tagValue = tagParts[1];
          var tagType = TagFilterType.values.firstWhere(
              (e) => e.toString() == tagName,
              orElse: () => TagFilterType.CATEGORIES);
          parametersList.add(TagFilter.fromType(tagFilterType: tagType, tagName: tagValue));
        }
      }
    }
  }

  ProductSearchQueryConfiguration configuration =
      ProductSearchQueryConfiguration(
    parametersList: parametersList,
    version: ProductQueryVersion.v3,
  );
  return OpenFoodAPIClient.searchProducts(
    User(userId: '', password: ''),
    configuration,
  );
}

void mkConfiguration() {
  OpenFoodAPIConfiguration.userAgent = UserAgent(
      name: 'dart_off_server',
      url: 'https://friendlyuser-dart-off-server.hf.space/');
  OpenFoodAPIConfiguration.globalLanguages = <OpenFoodFactsLanguage>[
    OpenFoodFactsLanguage.ENGLISH
  ];

  OpenFoodAPIConfiguration.globalCountry = OpenFoodFactsCountry.CANADA;
}

This code is written in Dart and makes use of the OpenFoodFacts API to search for food products. The code first checks for various parameters in the query map and adds the relevant filters to the parametersList array. These parameters include search terms, without additives, sort option, food category, page size and number, and tags.

The code then creates a ProductSearchQueryConfiguration object with the parameters in the parametersList and makes a call to the OpenFoodAPIClient's searchProducts method to search for products based on the configuration.

Finally, the code sets the OpenFoodAPIConfiguration user agent and global language and country.

 import 'package:openfoodfacts/openfoodfacts.dart';
 import 'package:dart_off_server/core.dart' as openfood;
 void main(List<String> arguments) async  { 
  ...
  openfood.mkConfiguration();
  final app = Alfred();
  app.get('/food/search', (req, res) { 
    final query = req.uri.queryParameters;
    return openfood.search(query).then((value) { 
      res.json(value); }
    ).catchError((error) { 
      res.json(error); 
    }); 
});

This Dart code defines a REST API endpoint for searching food information.

The first line imports the openfoodfacts library, which is used for accessing food product information through the OpenFoodFacts API.

The second line imports the dart_off_server library and creates an alias "openfood".

In the main function, the configuration for the OpenFoodFacts API is set up by calling openfood.mkConfiguration().

An instance of Alfred is then created and the API endpoint is defined with the line app.get('/food/search', (req, res) { ... }. The endpoint is listening for a GET request to the /food/search URL.

The req.uri.queryParameters object is used to extract the search parameters passed in the URL query string. The parameters are then passed to openfood.search(query) which performs the search.

The result of the search is returned as a JSON object to the client using res.json(value). If an error occurs during the search, it is caught and returned as a JSON object using res.json(error).

## References

https://friendlyuser-dart-off-server.hf.space/food/search?terms=%22Pizza%22 * https://huggingface.co/spaces/FriendlyUser/dart_off_server/tree/main

Scraping HTML Tables with Pandas

David Li — Mon, 20 Nov 2023 00:00:00 +0000

Here's an example of a blog post explaining how to scrape HTML tables using Pandas and read_html() method, along with a section on accessing class attributes in HTML elements:

Scraping HTML Tables with Pandas and read_html
Pandas is a powerful data analysis and manipulation library in Python, and one of its great features is the ability to easily extract data from HTML tables and transform them into DataFrames. One of the quickest ways to do this is using the read_html() method, which automatically converts HTML tables into a list of DataFrames.

Here's an example of how to use read_html() to extract data from a simple HTML table:

import pandas as pd

# Scrape the table from a webpage
tables = pd.read_html("https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States")

# The first table in the list is our desired table
df = tables[0]

# Display the first five rows of the DataFrame
print(df.head())

The read_html() method will search for all tables in the HTML content and return them as a list of DataFrames. In this example, we only want the first table, so we extract it using tables[0]. The resulting DataFrame will contain the data from the HTML table, with each row corresponding to a row in the HTML table and each column corresponding to a table cell.

Accessing Class Attributes in HTML Elements

In addition to extracting data from HTML tables, you may also want to extract information from specific HTML elements based on their class attributes. To do this, you can use a library such as BeautifulSoup, which provides a way to parse HTML and extract specific elements based on their attributes.

Here's an example of how to use BeautifulSoup to extract information from a table based on the class attribute of its cells:

import requests
from bs4 import BeautifulSoup

# Request the page
response = requests.get("https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States")

# Parse the HTML content
soup = BeautifulSoup(response.content, "html.parser")

# Find the table cells with the class attribute "flagicon"
flag_cells = soup.find_all("td", class_="flagicon")

# Extract the text from each cell
flags = [cell.text for cell in flag_cells]

# Print the extracted flags
print(flags)

In this example, we use requests.get() to retrieve the content of the webpage, and then parse it using BeautifulSoup. Next, we use the soup.find_all() method to find all td elements with a class attribute of "flagicon". Finally, we extract the text from each of these cells and store it in the flags list.

In conclusion, with Pandas and BeautifulSoup, you can easily extract data from HTML tables and specific elements based on their class attributes. These tools are incredibly useful for data analysis and manipulation, and with a bit of Python knowledge, you can quickly extract the information you need from webpages.

How I deployed a dart alfred server to hugging space

David Li — Mon, 13 Nov 2023 00:00:00 +0000

Docker is useful for deployments because it provides a consistent and reproducible environment for applications to run in, regardless of the host operating system. This makes it easier to deploy, test, and run applications in a variety of environments, from local development machines to production servers. Docker containers encapsulate an application and its dependencies, allowing for easy and reliable deployment, as well as efficient resource utilization. Additionally, Docker images can be stored in a central repository, making it easier to share and distribute applications.

Dart is a programming language developed by Google and it allows compiling Dart code into a single binary executable. This is achieved using the Dart SDK and the "dart2native" command-line tool, which compiles Dart code into machine code for a specific platform. The resulting binary is standalone and does not require a Dart runtime or interpreter to be installed on the target system. This makes it easy to distribute the application and eliminates the need for users to install any additional dependencies. Additionally, compiling to a single binary can result in faster startup times and improved performance compared to running the application in a Dart runtime.

 import 'package:dart_off_server/core.dart' as cli;
import 'package:alfred/alfred.dart';
import 'dart:io';

void main(List<String> arguments) async  {
  // get port from arguments
  var port = 7860;
  if (arguments.isEmpty) {
    print('Please provide a port number');
  } else {
    port = int.tryParse(arguments.first) ?? 6565;
  }
  final app = Alfred();

  // print line
  // print('Starting up server: on port $port');

  app.get('/', (req, res) => {'json_response': true});
  app.get('/text', (req, res) => 'Text response');

  app.get('/json', (req, res) => {'json_response': true});

  app.get('/jsonExpressStyle', (req, res) {
    res.json({'type': 'traditional_json_response'});
  });

  app.get('/file', (req, res) => File('test/files/image.jpg'));

  app.get('/html', (req, res) {
    res.headers.contentType = ContentType.html;
    return '<html><body><h1>Test HTML</h1></body></html>';
  });
   app.post('/post-route', (req, res) async {
    final body = await req.body; //JSON body
    body != null; //true
  });

  await app.listen(port); //Listening on port 6565
}

This code is a Dart program that sets up an HTTP server using the "alfred" package. The server listens on a specified port and provides several routes for handling HTTP requests. The routes include handling GET requests for JSON and HTML responses, serving a file, and handling a POST request. The port number is specified as the first argument when running the program, and if no argument is provided, the default port number is 6565. If an invalid argument is provided, the port number will be set to 6565. The code uses the Dart "async" keyword to indicate that the "listen" function is asynchronous and can run concurrently with other parts of the code.

Docker multi-stage builds are a feature in Docker that allow optimizing the size of the final Docker image by only including the necessary components and dependencies for production. Multi-stage builds involve using multiple "FROM" statements in the Dockerfile to create multiple intermediate images, each of which can be used as a temporary build environment.

For example, you can use one stage to build your application and its dependencies, and then use another stage to copy only the necessary files (e.g. the compiled binary) into a smaller, minimal base image, such as a runtime-only image. This reduces the size of the final image and minimizes the attack surface by eliminating unnecessary components.

Here's an example of a multi-stage build in a Dockerfile:

bash`# Build stage
FROM dart AS build
WORKDIR /app
COPY . .
RUN pub get && dart2native main.dart -o myapp

# Production stage
FROM alpine
WORKDIR /app
COPY --from=build /app/myapp /app
CMD ["./myapp"]

In this example, the first stage uses the dart image as the build environment and compiles the Dart code into a standalone binary. The second stage uses the alpine image as a minimal base image and copies only the compiled binary from the build stage into the final image. The final image only includes the necessary components to run the application, resulting in a smaller and more secure image.

 FROM dart:2.19.1 as builder
WORKDIR /app
COPY pubspec.yaml pubspec.lock ./
RUN dart pub get 
COPY . .
RUN dart compile exe bin/cli.dart -o /app/dart_off_server

# copy server file to basic image to run the app
FROM ubuntu:latest
WORKDIR /app
# RUN apk add --no-cache bash
COPY --from=builder /app/dart_off_server /app/dart_off_server
RUN ls -la
RUN pwd
CMD ["/app/dart_off_server"]

This Dockerfile creates two Docker images: a build environment image and a final image.

The first stage uses the dart:2.19.1 image as the base for the build environment and sets the working directory to /app. It then copies the pubspec.yaml and pubspec.lock files and runs dart pub get to install the dependencies. It then copies the rest of the application files and compiles the Dart code into an executable binary using the dart compile exe command. The resulting binary is saved as dart_off_server in the /app directory.

The second stage uses the ubuntu:latest image as the base for the final image and sets the working directory to /app. It then copies the dart_off_server binary from the build environment using the COPY --from=builder command. The final image only includes the necessary components to run the application, resulting in a smaller and more secure image.

The final step in the Dockerfile sets the command to run the application by executing the dart_off_server binary.

Ideally I would be able to use scratch of equivalent, but that didnt seem to work for hugging space, possibly for security reasons or image verification. Luckily ubuntu is about 30 MB so it is not too bad.
Dart is a modern, open-source, general-purpose programming language used for both front-end and back-end development. The advantages of using Dart for back-end development include:

Strong Typing: Dart has a strong type system, making it easier to write reliable code and catch bugs early in the development process.
Speed: Dart compiles to machine code, making it faster than some other interpreted languages. This can result in improved performance for back-end applications.
Scalability: Dart is designed for scalability, with features like isolate-based concurrency, making it well-suited for large-scale server-side applications.
Interoperability: Dart can interface with other technologies and platforms, such as C++ and JavaScript, making it possible to integrate with existing systems and technologies.
Ecosystem: Dart has a growing ecosystem of packages and tools, making it easier to find solutions to common back-end development problems.
Easy to Learn: Dart is a relatively new language and has a simple, straightforward syntax that makes it easy for developers to learn and start using it quickly.
Improved Developer Productivity: With its strong typing and other features, Dart can help developers write code faster and with fewer bugs, resulting in improved productivity and time-to-market for back-end applications.

References

To view the code base

Using golang to filter through reddit posts

David Li — Tue, 07 Nov 2023 00:00:00 +0000

Reddit is a social news aggregation, web content rating, and discussion website. Users can submit posts, such as text or direct links, and other users can vote on these posts, causing them to rise or fall in popularity. Reddit is organized into communities, known as "subreddits," which cover a vast array of topics and interests.

Web scraping with Go (Golang) can be useful in a variety of ways:

Data collection: Web scraping allows you to extract large amounts of data from websites, which can be useful for data analysis and research.
Price monitoring: Go can be used to monitor the prices of products on various websites and notify users of price changes.
Content aggregation: Go can be used to aggregate content from multiple websites into a single source, making it easier to find and view relevant information.
Automation: Go can be used to automate repetitive tasks, such as filling out forms or visiting websites, to save time and effort.
Testing: Go can be used to test websites for functional and performance issues, making it easier to identify and fix problems.

In summary, web scraping with Go is useful for data collection, price monitoring, content aggregation, automation, and testing, among other uses.

Parsing subreddits with the Reddit API involves using the API to extract data from Reddit and display it in a specific format. To parse subreddits using the Reddit API, you need to have an API key or access token. Once you have this, you can make HTTP requests to the Reddit API to retrieve data for a specific subreddit. The API will return the data in JSON format, which can then be processed and displayed in a way that meets your needs.

For example, to retrieve the top 10 posts in a subreddit, you can make a GET request to the following URL:

https://www.reddit.com/r/[subreddit]/top.json?limit=10

Where [subreddit] is the name of the subreddit you want to retrieve data from.

There are many libraries available in Go that simplify the process of making API calls and processing the resulting data. By using one of these libraries, you can focus on the data you want to extract and display, without having to worry about the details of making API calls and processing the results.

 package login 

import (
    "github.com/jzelinskie/geddit"
    "log"
    "github.com/dli-invest/finreddit/pkg/util"
)

// returns oauth session for reddit
// or fails - fine for my purposes
func RedditOAuth() (*geddit.OAuthSession, error) {
    client_id := util.GetEnvVar("REDDIT_CLIENT_ID")
    client_secret := util.GetEnvVar("REDDIT_CLIENT_SECRET")
    password := util.GetEnvVar("REDDIT_PASSWORD")
    username := util.GetEnvVar("REDDIT_USERNAME")
    o, err := geddit.NewOAuthSession(
        client_id,
        client_secret,
        "Stonk Market Scrapper see source https://github.com/dli-invest/finreddit",
        "http://friendlyuser.github.io",
    )
    if err != nil {
        log.Println(err)
        return nil, err
    }

    // Create new auth token for confidential clients (personal scripts/apps).
    err = o.LoginAuth(username, password)
    if err != nil {
        log.Println(err)
        return nil, err
    }
    return o, nil
}

This code defines a Go function named "RedditOAuth" that is used to authenticate a user to the Reddit API using the "geddit" library. The function starts by retrieving four environment variables ("REDDIT_CLIENT_ID", "REDDIT_CLIENT_SECRET", "REDDIT_PASSWORD", and "REDDIT_USERNAME") using the "util.GetEnvVar" function from the "github.com/dli-invest/finreddit/pkg/util" package.

Next, the function creates a new OAuth session using the "geddit.NewOAuthSession" function and the retrieved environment variables. If an error occurs while creating the OAuth session, it is logged and returned.

Finally, the function logs in the user using the "o.LoginAuth" method and the "username" and "password" environment variables. If an error occurs while logging in, it is logged and returned. If the login is successful, the OAuth session is returned.

 package reddit

import (
    "fmt"
    "log"
    "strings"
    "time"

    "github.com/dli-invest/finreddit/pkg/csvs"
    "github.com/dli-invest/finreddit/pkg/discord"
    "github.com/dli-invest/finreddit/pkg/login"
    "github.com/dli-invest/finreddit/pkg/types"
    "github.com/dli-invest/finreddit/pkg/util"
    "github.com/jzelinskie/geddit"
)

// gets submissions a given SRConfiguration
func GetSubmissions(session *geddit.OAuthSession, cfg types.SRConfig) []*geddit.Submission {
    subreddit := cfg.Name
    limit := cfg.Limit
    subOpts := geddit.ListingOptions{
        Limit: limit,
    }
    if cfg.After != "" {
        subOpts.After = cfg.After
    }
    submissions, err := session.SubredditSubmissions(subreddit, geddit.NewSubmissions, subOpts)
    if err != nil {
        log.Fatal("Failed to retrieve subreddit posts for " + subreddit)
    }
    // further filter entries by minScore and minComments
    var validSubmissions = []*geddit.Submission{}

    for _, submission := range submissions {
        if submission.NumComments != 0 && cfg.MinScore != 0 {
            if submission.NumComments >= cfg.MinComments && submission.Score >= cfg.MinScore {
                validSubmissions = append(validSubmissions, submission)
                continue
            }
        }
        if cfg.LinkFlairText != "" {
            // checking for flair
            if strings.Contains(submission.LinkFlairText, cfg.LinkFlairText) {
                validSubmissions = append(validSubmissions, submission)
                continue
            }
        }
        if len(cfg.Phrases) != 0 {
            // search through phrases
            title := strings.ToLower(submission.Title)
            // check matches word
            for _, phrase := range cfg.Phrases {
                // check if phrase is contained in title
                lowerPhrase := strings.ToLower(phrase)
                addSubmission := strings.Contains(title, lowerPhrase)
                if addSubmission {
                    validSubmissions = append(validSubmissions, submission)
                    continue
                }
            }
        }
    }
    return validSubmissions
}

// Scans subreddits from config file
// for example cmd/scan_sr/simple.yml
func ScanSRs(cfgPathStr string) {
    // login to reddit
    o, err := login.RedditOAuth()
    if err != nil {
        log.Fatal("Failed to initialize Reddit Scrapper")
    }
    // read subreddits from config file
    cfgPath := util.MkPathFromStr(cfgPathStr)
    cfg, err := util.NewConfig(cfgPath)
    if err != nil {
        log.Fatal(err)
    }
    csvsPath := util.MkPathFromStr(cfg.Data.CsvPath)
    // print header row only used when
    if cfg.Data.NoMessage {
        fmt.Printf("%s\t%s\t%s\t%s\t%s\t%s\n", "subreddit", "url", "title", "author", "linkFlairText", "date")
    }
    for _, srCfg := range cfg.Data.SubReddits {
        srSubmissions := GetSubmissions(o, srCfg)
        for _, s := range srSubmissions {
            // check if submission is in csv already
            // aware that constantly opening the csv is inefficient
            // but I am dealing with a reasonable amount of entires
            hasValue := csvs.FindInCsv(csvsPath, s.FullID, 1)
            if hasValue {
                // no value, do nothing here
            } else {
                // seems like a lot of posts, wondering if I will hit
                // post limit, sleep 2 seconds after each post.
                // append to csv
                sData := [][]string{{srCfg.Name, s.FullID, s.URL}}
                csvs.AppendToCsv(csvsPath, sData)
                if cfg.Data.NoMessage {
                    // output this as a csv for parsing in other programs
                    // fmt.Println("Not sending to subreddit")
                    fmt.Printf("%s\t%s\t%s\t%s\t%s\t%f\n", s.Subreddit, s.FullPermalink(), s.Title, s.Author, s.LinkFlairText, s.DateCreated)
                } else {
                    discordPayload := MapSubmissionToEmbed(s)
                    _, err := discord.SendWebhook(discordPayload)
                    if err != nil {
                        fmt.Println(s.FullID)
                        fmt.Println(err)
                    }
                    time.Sleep(2 * time.Second)
                }
            }
        }
    }
}

func MapSubmissionToEmbed(submission *geddit.Submission) types.DiscordPayload {
    description := fmt.Sprintf(
        "%s (%d Likes, %d Comments)",
        submission.Author,
        submission.Score,
        submission.NumComments)
    // get timestamp
    var dateCreated int64 = int64(submission.DateCreated)
    t := time.Unix(dateCreated, 0)
    timestamp := t.Format(time.RFC3339)
    title := fmt.Sprintf("%s - %s", submission.Subreddit, submission.Title)
    discordEmbed := []types.DiscordEmbed{{
        Title:       title,
        Url:         submission.URL,
        Description: description,
        Timestamp:   timestamp,
    }}
    discordPayload := types.DiscordPayload{Embeds: discordEmbed}
    return discordPayload
}

This code is a Reddit scraper in Golang, which is a part of a project "finreddit". It uses the geddit library for connecting to the Reddit API and retrieve submissions from given subreddit. The code filters the submissions based on various conditions such as minimum score, minimum comments, link flair text, and certain phrases. The filtered submissions are then appended to a CSV file. The code also includes a ScanSRs function, which reads subreddit configurations from a YAML file, logs into Reddit using OAuth authentication, and retrieves the submissions using the GetSubmissions function.

 package csvs

// csv utilties to prevent duplicate entries
// probably manually clear out every now and then
import (
    "encoding/csv"
    "fmt"
    "os"
)

// read csv from file path
func ReadCsvFile(filePath string) [][]string {
    f, err := os.Open(filePath)
    if err != nil {
        fmt.Println("Unable to read input file " + filePath, err)
    }
    defer f.Close()

    csvReader := csv.NewReader(f)
    records, err := csvReader.ReadAll()
    if err != nil {
        fmt.Println("Unable to parse file as CSV for " + filePath, err)
    }

    return records
}

func AppendRowToCsv(fileName string, data []string) {
    f, err := os.OpenFile(fileName, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
    if err != nil {
        fmt.Println(err)
        return
    }
    w := csv.NewWriter(f)
    err = w.Write(data)
    if err != nil {
        fmt.Println("Append Error")
        fmt.Println(err)
    }
    w.Flush()
}

// append rows to csvs
func AppendToCsv(fileName string, data [][]string) {
    f, err := os.OpenFile(fileName, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
    if err != nil {
        fmt.Println(err)
        return
    }
    w := csv.NewWriter(f)
    for _, row:= range data {
        err = w.Write(row)
        if err != nil {
            fmt.Println("Append Error")
            fmt.Println(err)
        }
    }
    w.Flush()
}

// check if value exists in csv
func FindInCsv(filePath string, searchValue string, searchColumn int) (bool) {
    records := ReadCsvFile(filePath)
    foundValue := false
    for _, row:= range records {
        valueInRow := row[searchColumn]
        if searchValue == valueInRow {
            foundValue = true
            break
        }
    }
    return foundValue
}

This is a Go library for working with CSV (Comma Separated Values) files. It provides functions to read, append and check if a value exists in a CSV file.

ReadCsvFile reads a CSV file from the given file path and returns the data as a two-dimensional slice of strings.
AppendRowToCsv appends a single row of data to the CSV file with the given file name.
AppendToCsv appends multiple rows of data to the CSV file with the given file name.
FindInCsv checks if a value exists in a specific column of the CSV file at the given file path. It returns a boolean indicating whether the value was found.

A Discord webhook is a tool that allows you to send messages to a Discord channel from an external source. It is a URL that enables you to send messages to a specific Discord channel from your own website, application, or other platform. The messages appear as if they were sent by a user within Discord, but they are actually sent programmatically. This allows you to integrate Discord into your own systems, such as sending notifications, updates, or other automated messages.

 package discord 

import (
    "os"
    "log"
    "encoding/json"
    "net/http"
    "github.com/dli-invest/finreddit/pkg/types"
    "bytes"
)

func SendWebhook(discordWebhook types.DiscordPayload) (*http.Response, error){
    discordUrl := os.Getenv("DISCORD_WEBHOOK")
    if discordUrl == "" {
        log.Fatal("DISCORD_WEBHOOK not set")
    }
    webhookData, err := json.Marshal(discordWebhook)
    if err != nil {
        log.Fatal(err)
    }
    resp, err := http.Post(discordUrl, "application/json", bytes.NewBuffer(webhookData))
    return resp, err
}

This is a Go function named "SendWebhook" that posts a Discord webhook to a specified Discord channel. The Discord webhook URL is obtained from an environment variable named "DISCORD_WEBHOOK". The input to the function is a DiscordPayload of type "types.DiscordPayload" and contains the data to be sent in the webhook. The function then uses the "json" package to marshal the input into a JSON string, and the "http" package to send a HTTP POST request to the Discord webhook URL with the JSON string as the payload. The response from the POST request and any error are returned from the function.

GitHub Actions allows you to automate tasks such as building, testing, and deploying code. It also supports automating tasks to run on a schedule using workflows. You can create a workflow that runs on a schedule using the "schedule" trigger in your workflow file.

Here's an example of a workflow that runs every day at 8pm UTC:

name: Schedule Task

on:
 schedule:
 - cron: '0 20 \* \* \*'

jobs:
 task:
 runs-on: ubuntu-latest
 steps:
 - name: Run Task
 run: echo "Running Task on Schedule"

This workflow specifies that it should run on a schedule using the schedule trigger and the cron value of 0 20 * * *. The cron value represents a standard cron syntax that specifies the schedule. In this case, it runs every day at 8pm UTC.

You can add additional steps in the steps section to perform any other tasks you need to automate. You can also use environment variables in your workflow to pass in any required information, such as API keys or file paths.

 on:
  schedule:
    # * is a special character in YAML so you have to quote this string
    - cron: '50 * * * *'
  push:
    branches:
      - main

env: 
  REDDIT_CLIENT_ID: ${{ secrets.REDDIT_CLIENT_ID }}
  REDDIT_CLIENT_SECRET: ${{ secrets.REDDIT_CLIENT_SECRET }}
  REDDIT_PASSWORD: ${{ secrets.REDDIT_PASSWORD }}
  REDDIT_USERNAME: ${{ secrets.REDDIT_USERNAME }}
  DISCORD_WEBHOOK: ${{ secrets.DISCORD_REDDIT_WEBHOOK}}

name: Send Updates
jobs:
  run:
    strategy:
      matrix:
        go-version: [1.15.x]
        platform: [ubuntu-latest]
    runs-on: ${{ matrix.platform }}
    steps:
    - name: Install Go
      if: success()
      uses: actions/setup-go@v2
      with:
        go-version: ${{ matrix.go-version }}
    - name: Checkout code
      uses: actions/checkout@v2
    - name: Run job
      run: go run cmd/scan_sr/manga.go

    - name: Commit files
      id: commit
      run: |
        git config --local user.email "action@github.com"
        git config --local user.name "GitHub Action"
        git add internal/*.csv
        git commit -m "Updating csv"
      continue-on-error: true

    - name: Push changes
      uses: ad-m/github-push-action@master
      continue-on-error: true
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        branch: ${{ github.ref }}

This code is a GitHub Actions workflow that automates the task of sending updates. The schedule of the task is set to run every hour at 50 minutes past the hour, and it will only run when the main branch is pushed.

The workflow sets environment variables for Reddit API credentials (client ID, client secret, password, and username) and a Discord webhook URL, all of which are stored as GitHub secrets.

The job runs on an Ubuntu latest machine and uses Go version 1.15.x. The job checks out the code, runs the scan_sr/manga.go script, and if successful, commits the updated csv files to the repository and pushes the changes. The action uses a GitHub push action to push the changes.

References

Reading indeed posts to scrap job listings (does not work anymore)

David Li — Wed, 25 Oct 2023 00:00:00 +0000

It is important to verify that humans are visiting a website because:

Preventing spam and bots: Verifying that only human visitors are accessing the site can help prevent spam and bots from artificially inflating traffic or spreading malicious content.
Protecting user data: By ensuring that only human visitors access the site, sensitive user information can be better protected from automated scraping and cyberattacks.
Improving user experience: Verifying human visitors can help improve the user experience by ensuring that the site's content and functionality is optimized for real people and not bots.
Measuring website performance: Accurately measuring website performance and user engagement requires accurate data about the number and behavior of human visitors.

In conclusion, verifying human visitors to a website is important for improving the site's security, performance, and user experience.

Cloudflare Protection is a suite of security features offered by Cloudflare, a web performance and security company. It provides a comprehensive set of tools and services to help protect websites from a wide range of online threats, including:

In conclusion, Cloudflare Protection offers a comprehensive set of security features that help protect websites from a wide range of online threats, improving website performance and security.

Since indeed has added cloudflare protection these scripts no longer work and the project has been open sourced.

 import bs4
import pandas as pd
import math
from datetime import datetime
from scrapper.fetch_data import get_xml
from scrapper.util import largest_number
def get_num_jobs(url_indeed)-> int:
  indeed_data = get_xml(url_indeed)
  job_string  = indeed_data.find("div", {"id": "searchCount"})
  job_count_indeed = job_string.contents[0:3]
  job_number_str = ''
  for item in job_count_indeed:
    if type(item) is bs4.element.Tag:
      job_number_str += item.get_text()
  if job_number_str.split()[-1] == 'jobs':
    print('Found jobs at the end')
    num_total_indeed = largest_number(job_number_str)
  else:
    print('Expected to find job number')
    assert(False)
  # 
  total_jobs = float(num_total_indeed)
  return int(math.ceil(total_jobs / 10.0))

# TODO Get a config file
def get_jobs_data(url_indeed, num_pages, default_location='Victoria, BC'):
  BASE_URL_indeed = 'https://ca.indeed.com'
  # create an empty dataframe
  job_df_indeed = pd.DataFrame()
  for i in range(1, num_pages+1):
    # generate the URL
    url = ''.join([url_indeed, '&start=', str(i*10)])
    print(url)

    # get the HTML code from the URL
    soup = get_xml(url)
    # pick out all the "div" with "class="job-row"
    divs = soup.findAll("div")
    job_divs = [jp for jp in divs if not jp.get('class') is None
                    and 'row' in jp.get('class')]

    # loop for each div chunk
    for job in job_divs:
      try:
        # job id
        id = job.get('data-jk', None)
        # job link related to job id
        link = BASE_URL_indeed + '/rc/clk?jk=' + id
        # job title
        title = job.find('a', attrs={'data-tn-element': 'jobTitle'}).attrs['title']
        # job company
        company = job.find('span', {'class': 'company'})
        if company is not None:
          company = company.text.strip()
        else:
          print('Company is missing')
        # job location
        location = job.find('span', {'class': 'location'})
        if location is not None:
          location = location.text.strip()
        else:
          location = default_location
          print('Location is missing')
      except Exception as e:
        print(e)
        continue

      job_df_indeed = job_df_indeed.append({
        'job_title': title,
        'job_company': company,
        'job_location':location,
        'job_link':link},
      ignore_index=True)
  cols=['job_title','job_company','job_location','job_link']
  job_df_indeed = job_df_indeed[cols]
  # delete the duplicated jobs using job link
  job_df_indeed = job_df_indeed.drop_duplicates(['job_link'], keep='first')
  return job_df_indeed

This code is a Python script that performs web scraping to collect job postings from the job search website Indeed. It uses the BeautifulSoup library to parse the HTML code of the pages and extract information such as job title, company, location, and link for each job posting.

The code first retrieves the number of job postings on the website by parsing the HTML code of the search results page and extracting the number of jobs found. It then uses that information to determine the number of pages of search results to retrieve.

Next, the code loops through each page of search results, extracts the relevant information for each job posting, and adds it to a pandas DataFrame. Finally, it returns the DataFrame containing all the job information, removing any duplicate job postings based on the job link.

 import pandas as pd
import re
from scrapper.fetch_data import get_xml

# Get an analyzer class, espeically if I have all these hardcoded values
types = ['Full-Time', 'Full Time', 'Part-Time', 'Part Time', 'Contract', 'Contractor']
type_lower = [s.lower() for s in types] # lowercases
# map the type_lower to type
type_map = pd.DataFrame({'raw': types, 'lower':type_lower}) # create a dataframe
type_map['raw'] = ["Full-Time", "Full-Time", 'Part-Time', 'Part-Time', "Contract", 'Contract'] # modify the mapping
type_dic = list(type_map.set_index('lower').to_dict().values()).pop() # use the dataframe to create a dictionary
# print(type_dic)

##### Skills #####
skills = ['R', 'Shiny', 'RStudio', 'Markdown', 'Latex', 'SparkR', 'D3', 'D3.js',
            'Unix', 'Linux', 'MySQL', 'Microsoft SQL server', 'SQL',
            'Python', 'SPSS', 'SAS', 'C++', 'C', 'C#','Matlab','Java',
            'JavaScript', 'HTML', 'HTML5', 'CSS', 'CSS3','PHP', 'Excel', 'Tableau',
            'AWS', 'Amazon Web Services ','Google Cloud Platform', 'GCP',
            'Microsoft Azure', 'Azure', 'Hadoop', 'Pig', 'Spark', 'ZooKeeper',
            'MapReduce', 'Map Reduce','Shark', 'Hive','Oozie', 'Flume', 'HBase', 'Cassandra',
            'NoSQL', 'MongoDB', 'GIS', 'Haskell', 'Scala', 'Ruby','Perl',
            'Mahout', 'Stata','Solidity']
skills_lower = [s.lower() for s in skills]# lowercases
skills_map = pd.DataFrame({'raw':skills, 'lower':skills_lower})# create a dataframe
skills_dic = list(skills_map.set_index('lower').to_dict().values()).pop()# use the dataframe to create a dictionary
# print(skills_dic)

##### Education #####
edu = ['Bachelor', "Bachelor's", 'BS', 'B.S', 'B.S.', 'Master', "Master's", 'Masters', 'M.S.', 'M.S', 'MS',
        'PhD', 'Ph.D.', "PhD's", 'MBA']
edu_lower = [s.lower() for s in edu]# lowercases
edu_map = pd.DataFrame({'raw':edu, 'lower':edu_lower})# create a dataframe
edu_dic = list(edu_map.set_index('lower').to_dict().values()).pop()# use the dataframe to create a dictionary
# print(edu_dic)

##### Major #####
major = ['Computer Science', 'Statistics', 'Mathematics', 'Math','Physics',
            'Machine Learning','Economics','Software Engineering', 'Engineering',
            'Information System', 'Quantitative Finance', 'Artificial Intelligence',
            'Biostatistics', 'Bioinformatics', 'Quantitative']
major_lower = [s.lower() for s in major]# lowercases
major_map = pd.DataFrame({'raw':major, 'lower':major_lower})# create a dataframe
major_dic = list(major_map.set_index('lower').to_dict().values()).pop()# use the dataframe to create a dictionary

##### Key Words ######
# For AI and BCHAIN, put the keywords here, as I want to get into those areas
keywords = ['Blockchain','Hashgraph','Ethereum','Solidity','Truffle','Drizzle','Vyper','Smart Contract',
'Machine Learning','Deep Learning','Data Science','Big Data',
'Web Analytics', 'Regression', 'Classification', 'User Experience', 
            'Streaming Data', 'Real-Time', 'Real Time', 'Time Series']
keywords_lower = [s.lower() for s in keywords]# lowercases
keywords_map = pd.DataFrame({'raw':keywords, 'lower':keywords_lower})# create a dataframe
keywords_dic = list(keywords_map.set_index('lower').to_dict().values()).pop()# use the dataframe to create a dictionary

def process_type(indeed_string, data_list, data_dict)-> list:
  type_matches = []
  for typ in data_list:
    if any(x in typ for x in ['+', '#', '.']):
      typp = re.escape(typ) # make it possible to find out 'c++', 'c#', 'd3.js' without errors
    else:
      typp = typ
    result = re.search(r'(?:^|(?<=\s))' + typp + r'(?=\s|$)', indeed_string) # search the string in a string
    if result:
      type_matches.append(data_dict[typ])
  return type_matches

def process_jobs(job_df_indeed):
  ##############################################
  ##### For Loop for scraping each job URL #####
  ##############################################
  # empty list to store details for all the jobs
  list_type = []
  list_skill = []
  # list_text = []
  list_edu = []
  list_major = []
  list_keywords = []

  for i in range(len(job_df_indeed)):
    try:
      # get the HTML code from the URL
      job_url = job_df_indeed.iloc[i]['job_link']
      soup = get_xml(job_url)
      # drop the chunks of 'script','style','head','title','[document]'
      for elem in soup.findAll(['script','style','head','title','[document]']):
        elem.extract()
      # get the lowercases of the texts
      texts = soup.getText(separator=' ').lower()

      # cleaning the text data
      job_text = re.sub(r'[\n\r\t]', ' ', texts) # remove "\n", "\r", "\t"
      job_text = re.sub(r'\,', ' ', job_text) # remove ","
      job_text = re.sub('/', ' ', job_text) # remove "/"
      job_text = re.sub(r'\(', ' ', job_text) # remove "("
      job_text = re.sub(r'\)', ' ', job_text) # remove ")"
      job_text = re.sub(' +',' ',job_text) # remove more than one space
      job_text = re.sub(r'r\s&\sd', ' ', job_text) # avoid picking 'r & d'
      job_text = re.sub(r'r&d', ' ', job_text) # avoid picking 'r&d'
      job_text = re.sub('\.\s+', ' ', job_text) # remove "." at the end of sentences

      # Job types
      list_type.append(process_type(job_text, type_lower, type_dic))

      # Skills
      list_skill.append(process_type(job_text, skills_lower, skills_dic))

      # Education
      list_edu.append(process_type(job_text, edu_lower, edu_dic))

      # Major
      list_major.append(process_type(job_text, major_lower, major_dic))

      # Key Words
      list_keywords.append(process_type(job_text, keywords_lower, keywords_dic))

      # Stop Words removal text
      # words = string.split(' ')
      # job_text = set(words) - set(stop_words) # drop stop words
      # list_text.append(list(job_text))
    except Exception as e:
      print(e)
      list_type.append(['Forbidden'])
      list_skill.append(['Forbidden'])
      list_edu.append(['Forbidden'])
      list_major.append(['Forbidden'])
      list_keywords.append(['Forbidden'])

  job_df_indeed['job_type'] = list_type
  job_df_indeed['job_skills'] = list_skill
  job_df_indeed['job_edu'] = list_edu
  job_df_indeed['job_major'] = list_major
  job_df_indeed['job_keywords'] = list_keywords

  return job_df_indeed

This is code for a Python function that pre-processes and maps various values related to job positions and skills to a standardized form. It defines and maps the following:

Job Types: ['Full-Time', 'Full Time', 'Part-Time', 'Part Time', 'Contract', 'Contractor']
Skills: ['R', 'Shiny', 'RStudio', 'Markdown', 'Latex', 'SparkR', 'D3', 'D3.js', 'Unix', 'Linux', 'MySQL', 'Microsoft SQL server', 'SQL', 'Python', 'SPSS', 'SAS', 'C++', 'C', 'C#','Matlab','Java', 'JavaScript', 'HTML', 'HTML5', 'CSS', 'CSS3','PHP', 'Excel', 'Tableau', 'AWS', 'Amazon Web Services ','Google Cloud Platform', 'GCP', 'Microsoft Azure', 'Azure', 'Hadoop', 'Pig', 'Spark', 'ZooKeeper', 'MapReduce', 'Map Reduce','Shark', 'Hive','Oozie', 'Flume', 'HBase', 'Cassandra', 'NoSQL', 'MongoDB', 'GIS', 'Haskell', 'Scala', 'Ruby','Perl', 'Mahout', 'Stata','Solidity']
Education: ['Bachelor', "Bachelor's", 'BS', 'B.S', 'B.S.', 'Master', "Master's", 'Masters', 'M.S.', 'M.S', 'MS', 'PhD', 'Ph.D.', "PhD's", 'MBA']
Major: ['Computer Science', 'Statistics', 'Mathematics', 'Math','Physics', 'Machine Learning','Economics','Software Engineering', 'Engineering', 'Information System', 'Quantitative Finance', 'Artificial Intelligence', 'Biostatistics', 'Bioinformatics', 'Quantitative']
Keywords: ['Blockchain','Hashgraph','Ethereum','Solidity','Truffle','Drizzle','Vyper','Smart Contract', 'Machine Learning','Deep Learning','Data Science','Big Data', 'Web Analytics', 'Regression', 'Classification', 'User Experience', 'Streaming Data', 'Real-Time', 'Real Time', 'Time Series']

The function "process_type" takes three parameters as input:

indeed_string: a string that needs to be processed
data_list: list of values
data_dict: dictionary of mapped values, created from data_list

The function uses regular expression (re) and looks for matches of the values in data_list in the indeed_string. If a match is found, it uses the corresponding mapping from data_dict to return the standardized form of the matched value.

Overall, this code is a component of a larger system to process job data fetched from a website using the get_xml function from the scrapper.fetch_data module.

 import os
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import base64
from mailjet_rest import Client
from scrapper.util import get_config
from ast import literal_eval
# TODO move to another package later
def generate_attachments(jobs_df):
  list_type = jobs_df['job_type'].tolist()
  list_skill = jobs_df['job_skills'].tolist()
  list_edu = jobs_df['job_edu'].tolist()
  list_keywords = jobs_df['job_keywords'].tolist()
  # implement later
  unique_data_type = set(x for l in list_type for x in l)
  unique_data_skills = set(x for l in list_skill for x in l)
  unique_data_edu = set(x for l in list_edu for x in l)
  unique_data_keywords = set(x for l in list_keywords for x in l)

  skills_dict={}
  for words in list_skill:
      for word in words:
          if not word in skills_dict:
              skills_dict[word]=1
          else:
              skills_dict[word]+=1
  # Calculate the frequency of keywords using a dataframe, consider adding plot or other data, perhaps save a csv and load it using d3
  result = pd.DataFrame()
  result['Skill'] = skills_dict.keys()
  result['Count'] = skills_dict.values()
  result['Ranking'] = result['Count']/float(len(list_skill))

  # Make another function
  results_dict={}
  for words in list_type:
      for word in words:
          if not word in results_dict:
              results_dict[word]=1
          else:
              results_dict[word]+=1
  # Calculate the frequency of keywords using a dataframe, consider adding plot or other data, perhaps save a csv and load it using d3
  result_type = pd.DataFrame()
  result_type['Type'] = results_dict.keys()
  result_type['Count'] = results_dict.values()
  result_type.set_index('Type') 

  results_dict={}
  for words in list_keywords:
      for word in words:
          if not word in results_dict:
              results_dict[word]=1
          else:
              results_dict[word]+=1
  # Calculate the frequency of keywords using a dataframe, consider adding plot or other data, perhaps save a csv and load it using d3
  result_keywords = pd.DataFrame()
  result_keywords['Keywords'] = results_dict.keys()
  result_keywords['Count'] = results_dict.values()

  result_keywords_html = result_keywords.to_html()

  matplotlib.use('Agg')
  skill_rank_plotname = 'skill-ranking.png'
  pie_plotname = 'pie.png'
  attachments_objs = []
  try:
    fig = plt.figure() # Create matplotlib figure
    default_size = fig.get_size_inches()
    ax = fig.add_subplot(111) # Create matplotlib axes
    ax2 = ax.twinx() # Create another axes that shares the same x-axis as ax.

    width = 0.4
    result.plot(x='Skill', y='Count', kind='bar', table=False, yerr=None,ax=ax, width=width, position=1,legend=False)
    result.plot(x='Skill', y='Ranking', kind='bar',table=False, yerr=None,ax=ax2, width=width, position=0,legend=False)

    ax.set_ylabel('Count')
    ax2.set_ylabel('Ranking')
    fig.set_size_inches((default_size[0]*2, default_size[1]*2))
    fig.tight_layout()
    ### This should be 2 x 2 times bigger, one 2 for dpi and one 2 for default size
    plt.savefig(skill_rank_plotname, dpi = 200)
  except Exception as e:
    print(e)
    print('Failed to make image for skill_rank')

  # On gitlab if there is a space in the file name, it doesn't send properly
  try:
    result_type.plot.pie(y='Count', labels=result_type['Type'], autopct='%1.1f%%', figsize=(5, 5))
    plt.savefig(pie_plotname, dpi = 200)
  except:
    print('Failed to make pie chart for type')
  for index, plot_file in enumerate([pie_plotname, skill_rank_plotname]):
    print(index)
    with open(plot_file, "rb") as img_file:
      base64_image = base64.b64encode(img_file.read())
    attachments_objs.append(
      {
        "ContentType": "image/png",
        "Filename": f"{plot_file}",
        "ContentID": f"id{index}",
        "Base64Content": base64_image.decode('ascii')
      }
    )
  return attachments_objs

def send_mailjet_email(jobs_df, metadata):
  api_key = os.environ['MJ_APIKEY_PUBLIC']
  api_secret = os.environ['MJ_APIKEY_PRIVATE']
  mailjet = Client(auth=(api_key, api_secret), version='v3.1')
  cfg = get_config()
  get_attachments = generate_attachments(jobs_df)
  keywords_skills = """\
    <div>
      <h4>Graphs</h4>
        <img src="cid:id0" alt = "Plot not generated" style="width:100%"/>
        <img src="cid:id1" alt = "Plot not made" style="width:100%"/>
      <h5> Potential Jobs </h5>
      {jobsTable}
    </div>
  """.format(jobsTable=jobs_df.to_html())
  data = {
    'Messages': [
      {
        "From": {
          "Email": cfg['email']['from'],
          "Name": "Indeed Scrapper"
        },
        "To": [
          {
            "Email": cfg['email']['to'],
            "Name": "Reciever Email"
          }
        ],
        "Subject": "Job Report",
        "HTMLPart": keywords_skills,
        "InlinedAttachments": get_attachments
      }
    ]
  }
  result = mailjet.send.create(data=data)
  print(result.status_code)
  print(result.json())

This is a function to generate data visualization and email attachments. Given a dataframe of job information, the function first extracts information of job types, skills, education, and keywords into separate lists and computes the frequency of each item.

For job skills, the function uses a dictionary to store the count of each skill, converts the dictionary into a pandas dataframe, and adds a new column for the ranking of each skill. It then generates a bar plot with two y-axes showing the count and ranking of each skill and saves it as an image.

For job types, the function computes the frequency of each type and converts it into a pandas dataframe. It then generates a pie chart from the dataframe and saves it as an image.

Finally, the function converts the frequency of keywords into a pandas dataframe and converts it into HTML. It also opens the two saved images and converts them into email attachments.

To avoid cloudflare blocking the requests, you can use selenium, but you may need a workaround to bypass the captcha / human verification.

To scrape a site using selenium and python, you need to perform the following steps:

Install selenium: You can install selenium by running pip install selenium in your terminal or command prompt.
Download the web driver: Selenium requires a web driver to interact with the browser. You can download the web driver for the browser of your choice from the official Selenium website.
Import selenium in your python script: Add the following code to import selenium in your python script: from selenium import webdriver.
Initialize the web driver: In your script, initialize the web driver by adding the following code:

makefile`browser = webdriver.Firefox() # for firefox browser
# or
browser = webdriver.Chrome() # for chrome browser

Navigate to the website: Use the following code to navigate to the website you want to scrape: browser.get("https://www.example.com"). Replace "https://www.example.com" with the URL of the website you want to scrape.
Find elements: To find elements on the website, you can use various methods provided by the selenium library such as find_element_by_id, find_element_by_name, find_element_by_class_name, find_element_by_xpath, etc.
Extract information: Once you have found the element, you can extract the information from it. For example, to extract text from a tag, you can use the following code: element.text.
Close the browser: Finally, close the browser using the following code: browser.quit().

Here's a basic example of how to scrape a site using selenium and python:

scss`from selenium import webdriver

browser = webdriver.Firefox()
browser.get("https://www.example.com")

element = browser.find\_element\_by\_xpath("//h1")
print(element.text)

browser.quit()

This code will open a firefox browser, navigate to the example.com website, find the first <h1> tag on the page, and print its text.

Converting React CSS styles to CSS classes

David Li — Fri, 20 Oct 2023 13:00:00 +0000

Converting React CSS styles to CSS classes is important for several reasons:

Separation of Concerns: By using CSS classes, the styling and layout of a React component can be separated from its functionality. This makes the code easier to maintain and update, as changes to the styling don't affect the component's behavior.

Reusability: CSS classes can be reused across multiple components, making it easier to maintain consistency in the appearance of a website or application.

Performance: Inlining styles in a React component can lead to slow performance, as the styles need to be recalculated every time the component is re-rendered. Using CSS classes instead can improve the performance, as the styles are cached by the browser.

Better Tooling: CSS class-based styles can be optimized and minified using tools such as CSS preprocessors and postprocessors, making the code more efficient and faster to load.

For example the following inline styles can be posted to chatgpt

  border: "1px solid #fff",
  paddingRight: "4px",
  paddingLeft: "4px",
  paddingTop: "5px",
  paddingBottom: "5px",
  minWidth: "75px",
  backgroundColor: "white",
  boxShadow: "0 0 2px 0 rgba(0, 0, 0, 0.5)",
  fontSize: "14px",
  textAlign: "center",

Pasting that to chatgpt, it gives me this:

.some-class {
  border: 1px solid #fff;
  padding-right: 4px;
  padding-left: 4px;
  padding-top: 5px;
  padding-bottom: 5px;
  min-width: 75px;
  background-color: white;
  box-shadow: 0 0 2px 0 rgba(0, 0, 0, 0.5);
  font-size: 14px;
  text-align: center;
}

This illustrates how simple it can be to build useful tools that boost developer productivity and make life easier for everyone.

In conclusion, converting React CSS styles to CSS classes is a best practice that can lead to better maintainability, performance, and overall organization of code.

Liability Driven Investing flow chart

David Li — Fri, 13 Oct 2023 00:00:00 +0000

Liability Driven Investing (LDI) is a portfolio management strategy where the investment decisions are driven by the specific needs of the investor's liabilities (e.g. pension obligations). The goal is to match the duration, currency, and risk profile of the assets with the liabilities, to minimize the risk of not being able to meet future obligations.

The risk of high leverage in investing is that it amplifies potential gains, but also increases the risk of losses. If the invested assets underperform, the leveraged positions can result in large losses that exceed the initial investment. Additionally, leveraged investments can be subject to margin calls and other forms of forced selling, which can amplify losses in declining markets. High leverage can also increase the risk of bankruptcy or financial distress.

 \documentclass{standalone}
\usepackage[utf8]{inputenc}
\usepackage{tikz}
\usetikzlibrary{positioning}
\usepackage{xcolor}
\begin{document}
\begin{tikzpicture}[node distance=5cm]

\tikzset{
    connector/.style={
        -latex,
        font=\scriptsize
    },
    connectorReverse/.style={
        -stealth,
        font=\scriptsize
    },
    rectangle connector/.style={
        connector,
        to path={(\tikztostart) -- ++(#1,0pt) \tikztonodes |- (\tikztotarget) },
        pos=0.5
    },
    rectangle connector/.default=-2cm,
    rectangleReverse connector/.style={
        connector,
        to path={(\tikztotarget) -- ++(#1,0pt) \tikztonodes |- (\tikztostart) },
        pos=0.75
    },
    rectangleReverse  connector/.default=-2cm,
    straight connector/.style={
        connector,
        to path=--(\tikztotarget) \tikztonodes
    }
}
  \node (company) [shape=rectangle, draw, fill=blue!5, align=left, text width = 7cm,  inner sep=1cm] {
    \textbf{Company} \\
    Employers promise to pay their staff a pension
  };
  \node (pension) [shape=rectangle, draw, fill=blue!5, below of=company, text width = 7cm, align=left, inner sep=1cm] {
  \textbf{Defined Benefit Pension Scheme} \\
    A workplace pension based  on salary and term of employment, rather than how much is paid in. These schemes are increasing rare.
  };
  \node (asset) [shape=rectangle, draw, fill=blue!5, below of=pension,  text width = 7cm, align=left, inner sep=1cm] { \textbf{Asset Manager} \\
  Appointed by pension scheme trustees to manage assets on their behalf} ;
  \node (fund) [shape=rectangle, draw, fill=blue!5, below of=asset,  text width = 7cm, align=left, inner sep=1cm] { \textbf{Liability Driven Investment Fund} \\ 
  A type of fund used by the asset manager to invest the pension scheme's money.
  };
  \draw [->] (company) -- (pension);
  \draw [->] (pension) -- (asset);
  \draw [->] (asset) -- (fund);


   \node (equities) [shape=rectangle, draw, fill=red!5, below left=3cm and 2cm of fund, text width = 2cm, align=left, inner sep=1cm] { \textbf{Equities}};
  \node (bonds) [shape=rectangle, draw, fill=red!5, right of=equities, text width = 2cm, align=left, inner sep=1cm] { \textbf{Bonds}};
  \node (gilts) [shape=rectangle, draw, fill=red!5, right of=bonds, text width = 2cm, align=left, inner sep=1cm] { \textbf{Gilts}};
  \node (derivatives) [shape=rectangle, draw, fill=red!5, below, right of=gilts, text width = 2cm, align=left, inner sep=1cm] { \textbf{Derivatives}};

  \node [above left=2cm and -5.5cm of equities, align=left, text width=2cm] {Funds can invest in a variety of assets};
  \node (fwaypoint)[draw=none, below =1.75cm of fund] {};
  % \draw [-] (fund.south) to (fwaypoint);
  \draw [-stealth, rectangleReverse connector=2cm] (fund.west) to (equities.north west);
  \draw [-stealth, rectangleReverse connector=2cm] (fund.east) to (derivatives.north west);
  \draw [->] (bonds.north) to (fund);
   \draw [->] (gilts.north) to (fund);
\end{tikzpicture}

\end{document}

This is a diagram that demonstrates how the ldi (liability driven investing) is intended to work. I personally do not believe in ldi investing.

Flow charts are important because they help to:

Visualize and understand complex processes: Flow charts help to break down a complex process into simple and easily understandable steps.
Communicate ideas effectively: Flow charts are a visual representation of information, making it easier to convey ideas to others and collaborate effectively.
Improve efficiency: By mapping out processes and identifying areas for improvement, flow charts can help organizations streamline their workflows and make them more efficient.
Track progress: Flow charts allow you to see the progress of a project or process and make adjustments if needed.
Identify bottlenecks: Flow charts help to identify bottlenecks or areas of a process where improvement is needed.

Overall, flow charts play a crucial role in improving communication, making processes more efficient, and helping to achieve desired outcomes in various settings.

Investing homepage with waves animation

David Li — Thu, 05 Oct 2023 00:00:00 +0000

An investing homepage can provide a central location for an individual or organization to access financial market information, research, and tools to make informed investment decisions. It can also serve as a hub for managing a portfolio, tracking investments, and monitoring performance. Additionally, a homepage can provide resources such as educational materials, news updates, and access to financial advisors or other professionals for guidance. Overall, an investing homepage can be a useful tool for individuals and organizations to stay informed and make strategic decisions about their investments.

Here is an example of a wave animation using Vue.js:

html`<template>
 <div>
 <svg width="100%" height="100%" viewBox="0 0 1440 320">
 <path
 v-for="(item, index) in waves"
 :key="index"
 :d="item"
 :fill="fill"
 />
 </svg>
 </div>
</template>

<script>
export default {
 data() {
 return {
 waves: [],
 fill: '#f8bbd0'
 }
 },
 mounted() {
 for (let i = 0; i < 3; i++) {
 this.waves.push(this.drawWave(i \* 480))
 }
 },
 methods: {
 drawWave(shift) {
 let data = ''
 for (let x = 0; x <= 1440; x++) {
 let y = Math.sin((x + shift) / 30) \* 50
 if (x === 0) {
 data += `M ${x} ${160 + y}
 } else {
 data += ` L ${x} ${160 + y}
 }
 }
 return data + ' Z'
 }
 }
}
</script>

This code is using v-for to loop through an array of wave paths and render them as SVG paths. The drawWave method is used to generate the d attribute for each path, which represents the shape of the wave. The shift variable is used to offset the position of each wave and create a sense of movement. The fill variable is used to set the color of the wave.

You can customize this code to suit your needs, such as adjusting the amplitude, frequency, or color of the waves. Also you can use CSS to style the SVG as you want.

 <template>
  <div>
    <div class="wave-container">
      <div class="hero-container">
        <div class="grid hero-grid">
          <vs-row justify="center">
            <vs-col xs="12" lg="6">
              <div class='hero-body'>
                <h2>Personal Investment</h2>
                <h3>Blog</h3>
                <p>Providing free open source tools and writing down my investment thoughts.</p>
              </div>
            </vs-col>
            <vs-col xs="12" lg="6">
              <div class="hero-img-container">
                <img class='hero-image' src="~/static/undraw_investing_7u74.svg" />
              </div>
            </vs-col>
          </vs-row>
        </div>
      </div>
      <div class='wave-svg'>
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1440 320">
          <path fill="#5b3cc4" fill-opacity="1" d="M0,320L21.8,293.3C43.6,267,87,213,131,213.3C174.5,213,218,267,262,288C305.5,309,349,299,393,282.7C436.4,267,480,245,524,229.3C567.3,213,611,203,655,202.7C698.2,203,742,213,785,229.3C829.1,245,873,267,916,261.3C960,256,1004,224,1047,218.7C1090.9,213,1135,235,1178,218.7C1221.8,203,1265,149,1309,160C1352.7,171,1396,245,1418,282.7L1440,320L1440,0L1418.2,0C1396.4,0,1353,0,1309,0C1265.5,0,1222,0,1178,0C1134.5,0,1091,0,1047,0C1003.6,0,960,0,916,0C872.7,0,829,0,785,0C741.8,0,698,0,655,0C610.9,0,567,0,524,0C480,0,436,0,393,0C349.1,0,305,0,262,0C218.2,0,175,0,131,0C87.3,0,44,0,22,0L0,0Z"></path>
        </svg>
      </div>
    </div>
  </div>
</template>

<script>
import Logo from '~/components/Logo.vue'
import VuesaxLogo from '~/components/VuesaxLogo.vue'

export default {
  components: {
    Logo,
    VuesaxLogo
  },
  mounted() {
    this.$nextTick(() => {
      this.$nuxt.$loading.start()

      setTimeout(() => this.$nuxt.$loading.finish(), 5000)
    })
  }
}
</script>

<style>

.hero-grid {
  z-index: 5;
}
.hero-img-container {
  max-width: 350px;
  max-height: 220px;
  margin: auto;
}

.hero-body {
  margin: auto;
  padding: 1px 6px;
  text-align: right;
}
.hero-image {
  max-width: 100%;
}
@media (min-width: 768px) and (max-width: 979px) {
  .hero-image {
    right: 100px;
  }
}

.hero-container {
  margin: 0 auto;
  display: flex;
  justify-content: center;
  align-items: center;
  text-align: center;
  padding-bottom: 25px;
  margin-bottom: -1px;
  z-index: 10;
}
@media (min-width: 769px) {
  .hero-container {
    margin-top: 125px;
  }
}
@media (max-width: 768px) {
  .hero-container {
    margin-top: 100px;
  }
  .img-container {
    margin-top: 50px;
  }
}
.container {
  margin: 0 auto;
  min-height: 100vh;
  display: flex;
  justify-content: center;
  align-items: center;
  text-align: center;
}

.title {
  font-family:
    'Quicksand',
    'Source Sans Pro',
    -apple-system,
    BlinkMacSystemFont,
    'Segoe UI',
    Roboto,
    'Helvetica Neue',
    Arial,
    sans-serif;
  display: block;
  font-weight: 300;
  font-size: 55px;
  color: #35495e;
  letter-spacing: 1px;
  text-transform: capitalize;
  margin: 25px 0;
}

.subtitle {
  font-weight: 300;
  font-size: 1.1rem;
  color: #526488;
  word-spacing: 2px;
  padding-bottom: 15px;
  max-width: 600px;
}

.subtitle a {
  font-weight: 500;
  color: inherit;
}

.links {
  padding-top: 15px;
  margin-bottom: 20px;
}

.content-logos {
  display: flex;
  align-items: center;
  justify-content: center;
  min-width: 500px;
}

.plus {
  font-size: 2.5rem;
  margin: 15px;
  color: #35495e;
}

.h3 {
  font-family:
    'Quicksand',
    'Source Sans Pro',
    -apple-system,
    BlinkMacSystemFont,
    'Segoe UI',
    Roboto,
    'Helvetica Neue',
    Arial,
    sans-serif;
  font-weight: 400;
  margin: 10px;
}

.wave-container {
  background: #5b3cc4;
  color: #FFF;
  text-align: center;
  overflow: hidden;
}

.wave-svg {
  position: relative;
  background-color: white;
  display: block;
  transform-origin: bottom;
  margin-top: -100px;
  z-index: 1;
}

@keyframes wave {
  0% {
    margin-left: 0;
  }
  10% {
    margin-left: -200px;
  }
  40% {
    margin-left: -400px;
  }
  60% {
    margin-left: -200px;
  }
  75% {
    margin-left: -300px;
  }
  95% {
    margin-left: -50px;
  }
  100% {
    margin-left: -5px;
  }
}


.wave-svg > svg:nth-of-type(1) {
  animation: wave 7s cubic-bezier( 0.36, 0.45, 0.63, 0.53) infinite;
}


</style>

This is an example of how to use Vue.js to create a wave animation as part of a larger webpage. In this example, the wave animation is implemented using an SVG path that is styled with CSS. The wave is contained within a "wave-svg" div and is positioned below a "hero-container" div that contains the main content of the page. The "hero-container" div includes a grid layout using Vuesax components (vs-row and vs-col) to arrange the content, including a heading, a subtitle, and an image. The example also includes some media queries to adjust the layout for different screen sizes. Finally, the example uses a mounted() lifecycle method to add a loading spinner that starts when the page loads and finishes after 5 seconds.

This code is an example of how to use Vue.js and the Vuesax library to create a wave animation that can be used in a website or web application. The animation is created using an SVG path element that is styled with CSS and rendered on the page using Vue template syntax. The wave-svg class is applied to the SVG element which contains the path element, which is the element that creates the wave animation. The animation itself is done using the CSS fill property and the fill-opacity property to create the wave effect.

The code also makes use of Vuesax's grid system to create a responsive layout for the wave animation and other elements on the page, such as an image and some text. Additionally, the code uses a mounted lifecycle hook to show a loading spinner for 5 seconds before the wave animation is shown, this can be removed if not necessary.

Natural Gas Price Analysis with Python

David Li — Wed, 27 Sep 2023 00:00:00 +0000

Gas prices fluctuate due to a variety of factors, including changes in the cost of crude oil, which is the primary raw material used to produce gasoline, as well as changes in supply and demand for gasoline, taxes, and other costs associated with refining, storing, and transporting gasoline. Additionally, geopolitical events, such as conflicts or natural disasters, can also affect gas prices by disrupting the global oil market.

Cyclical stocks are stocks that tend to be sensitive to the economic cycle, meaning their prices tend to rise and fall with the overall health of the economy. These stocks are often associated with industries that are heavily influenced by consumer spending, such as retail, consumer durables, and homebuilding.

Cyclical stocks tend to perform well during economic expansions and periods of low unemployment, as consumers have more disposable income to spend on non-essential goods and services. However, during economic downturns and recessions, when consumer spending decreases, the profits and stock prices of companies in cyclical industries tend to decline.

Examples of cyclical stocks include those in the automotive, construction, housing, and consumer discretionary sectors.

A 3-month moving average is a technical indicator that is used to smooth out short-term fluctuations in a stock or index's price, and to help identify longer-term trends. It is calculated by taking the average of a stock or index's closing price over the past three months.

The idea behind a moving average is that it helps to filter out "noise" in the stock or index's price, and to make it easier to spot trends. By averaging out the closing prices over a longer period of time, such as three months, the moving average smooths out any short-term volatility in the stock or index's price, and makes it easier to identify longer-term trends.

Traders often use moving averages to help identify buy and sell signals. For example, if a stock or index's price is above its 3-month moving average, it may be considered to be in an uptrend, and traders may buy the stock or index. Conversely, if a stock or index's price is below its 3-month moving average, it may be considered to be in a downtrend, and traders may sell the stock or index.

 # calculate average from daily data from natural gas futures from yahoo finance
import yfinance
import pandas as pd
import mplfinance as mpf

# read NG=F 2 years worth of data

nat_gas = yfinance.download("NG=F", start="2021-01-01", end="2023-01-01")

nat_gas['Month'] = pd.to_datetime(nat_gas.index)
# pad all date
# print(nat_gas.groupby(by=['Volume', 'Adj Close', pd.Grouper(key='Month', freq='3M')])['Open'].agg('mean'))

mths_3 = pd.date_range(end = nat_gas.index[-1], freq=pd.offsets.BQuarterEnd(),periods=8)
# mths_6 = pd.date_range(end = nat_gas.index[-1], freq=pd.offsets.MonthBegin(6),periods=4)

mth3 = (nat_gas
         #filter for last three months
        .loc[mths_3]
        .groupby(["Month", "Volume"])
        .agg(avg_ng_last_3_months=("Open","mean"))
        )

print(mth3)

# for Bir.to (birchcliff energy, I use the following numbers for some estimations)
# random guess from last two quarters of profit
# at 4.558 assume 360 revenue for Q4 and 160M profit
# 475 revenue and 260M profit for Q4
# considering the investor presentation, thinking its closer to 360M revenue and 160 M profit.
quarterly_profit = 160
#   266.05M * 0.2
# if deliveries are 3 quarters behind then 


quarterly_dividend = 266.05 * 0.2
print(quarterly_dividend)

remaining_revenue = quarterly_profit - quarterly_dividend
print(remaining_revenue)
# 176 debt
total_debt = 176

# max debt reduction (they target to get to 50M USD in debt)
min_debt_reduction = total_debt - 100

print(min_debt_reduction)
# bunch of risk of falling gas prices, but with new supply reduction from russia, doubt it would be so bad.

# save fig
mpf.plot(nat_gas,type='ohlc',mav=(30, 60, 90), volume=True, style='yahoo', savefig='ng.png')

This script appears to be using the yfinance library to download historical data for natural gas futures (NG=F) from Yahoo Finance, for the period between January 1, 2021 and January 1, 2023. It then uses the Pandas library to create a new column in the dataframe called "Month" which is populated with the date of the data point, and converts it to a datetime object.

It then uses the Pandas library to create a date range for the last 8 quarters, and filters the dataframe to only include data points from those quarters. It then groups the data by the "Month" column and the "Volume" column, and calculates the average "Open" value for each group, which it saves to a new column in the dataframe called "avg_ng_last_3_months".

It also appears to be doing some calculations related to a company's profit, dividend, and debt, but it's not clear how these calculations relate to the data being downloaded or how they are being used.

After that it is using mplfinance library to plot Open high low and close prices of natural gas with 30, 60 and 90 moving averages, and volume, and then it's saving the graph with the name ng.png.

There are several libraries in Python that can be used to create financial plots, including:

Matplotlib: A widely-used library for creating static, animated, and interactive visualizations in Python. It can be used to create a wide variety of plots, including line plots, scatter plots, bar plots, and candlestick plots.
Plotly: A library for creating interactive visualizations in Python. It can be used to create a wide variety of plots, including line plots, scatter plots, bar plots, and candlestick plots.
mplfinance: A library built on top of matplotlib that is specifically designed for financial plotting. It can be used to create a wide variety of financial plots, including candlestick plots, barcharts and volume plots.
seaborn: A library for creating statistical visualizations in Python. It can be used to create a wide variety of plots, including line plots, scatter plots, bar plots, and heatmaps.
bokeh: A library for creating interactive visualizations in Python. It can be used to create a wide variety of plots, including line plots, scatter plots, bar plots, and candlestick plots.

All of these libraries have their own strengths and weaknesses, and the best choice will depend on the specific needs of the project. Some are better for creating static plots, while others are better for creating interactive visualizations. Some are better for working with large datasets, while others are better for creating highly customized visualizations.

Most of the libraries mentioned above require a basic understanding of the plotting and visualization concepts, Some of the libraries like mplfinance are built on top of matplotlib and are more specialized to financial plots, making it easier to create charts like candlestick charts and other financial plots.

Climate change is expected to have a significant impact on gas consumption as the world shifts towards cleaner and more sustainable energy sources. As temperatures rise and weather patterns become more extreme, the demand for natural gas is expected to increase in certain sectors such as power generation, as it is considered a cleaner alternative to coal and oil. However, in other sectors such as transportation, the demand for natural gas may decrease as electric vehicles and other alternative fuel vehicles become more prevalent.

The use of natural gas for power generation is expected to increase as countries look to phase out coal and reduce their greenhouse gas emissions. Natural gas is a less carbon-intensive fossil fuel and can be used in combination with renewable energy sources such as wind and solar power to provide a flexible and reliable source of electricity.

However, the transportation sector is also a significant contributor to global greenhouse gas emissions, and there is a growing movement to shift away from fossil fuels and towards electric and alternative fuel vehicles. As a result, the demand for natural gas as a transportation fuel is expected to decrease in the long term.

Overall, the impact of climate change on gas consumption is complex and will depend on a variety of factors, including government policies, technological advancements, and consumer preferences. It's important to note that natural gas can be used as a transition fuel to decrease the use of coal and oil in power production and make the transition to renewable energy sources.

Using chatgpt to generate blog posts

David Li — Wed, 20 Sep 2023 00:00:00 +0000

ChatGPT is a large language model that has been trained using a technique called unsupervised learning, in which the model learns to generate text by being fed a large dataset of text. It is designed to generate human-like responses to prompts, and can be used for a variety of natural language processing tasks, such as language translation, text summarization, and question answering.

 import os
import re
import argparse
from pyChatGPT import ChatGPT
# import yaml
from yaml import load, dump, Loader
import io
import warnings
from PIL import Image
from stability_sdk import client
import stability_sdk.interfaces.gooseai.generation.generation_pb2 as generation
# Our Host URL should not be prepended with "https" nor should it have a trailing slash.
os.environ['STABILITY_HOST'] = 'grpc.stability.ai:443'
os.environ["STABILITY_KEY"] = os.getenv("STABILITY_KEY")

def generate_image(cfg:dict)-> None:
    prompt = cfg['imageArgs']['prompt']
    stability_api = client.StabilityInference(
        key=os.environ['STABILITY_KEY'], # API Key reference.
        verbose=True, # Print debug messages.
        engine="stable-diffusion-v1-5", # Set the engine to use for generation. 
        # Available engines: stable-diffusion-v1 stable-diffusion-v1-5 stable-diffusion-512-v2-0 stable-diffusion-768-v2-0 
        # stable-diffusion-512-v2-1 stable-diffusion-768-v2-1 stable-inpainting-v1-0 stable-inpainting-512-v2-0
    )
    answers = stability_api.generate(
        prompt=prompt
    )
    for resp in answers:
        for artifact in resp.artifacts:
            if artifact.finish_reason == generation.FILTER:
                warnings.warn(
                    "Your request activated the API's safety filters and could not be processed."
                    "Please modify the prompt and try again.")
            if artifact.type == generation.ARTIFACT_IMAGE:
                img = Image.open(io.BytesIO(artifact.binary))
                # images
                img.save("images" + "/" + str(artifact.seed)+ ".png") # Save our generated images with their seed number as the filename.

def generate_frontmatter(cfg: dict)-> None:
    # open output file
    output_file = cfg['outputFile']
    with open(output_file, 'w') as f:
        # write frontmatter
        f.write('---\n')
        # get frontmatter
        frontmatter = cfg['frontMatter']
        for key, value in frontmatter.items():
            f.write(f'{key}: {value}\n')
        f.write('---\n')
        # write body
    pass

def use_programming_language(cfg: dict, section_text: str)-> None:
    # get programming language
    programming_language = cfg['programmingLanguage']
    # use regex to find plain ```
{% endraw %}
 and replace with
{% raw %}
 ```programming_language
    type_start_line = None
    type_end_line = None
    # replace with code blocks with programming language
    # split lines by \n
    modified_lines = []
    lines = section_text.split('\n')
    for i, line in enumerate(lines):
        if line == '```

':
            if type_start_line is None:
                type_start_line = i
                # append modified line
                modified_lines.append(f'
{% raw %}
```{programming_language}')
            else:
                # odd number of ```
{% endraw %}
 so we are ending a code block
                type_start_line = None
                # append modified line
                modified_lines.append('

```')
        else:
            # adjust line if type_start_line is not None
            # remove ` at the beginning of the line and the end of the line
            if type_start_line is not None:
                # check for ` character at the beginning of the line
                if line[0] == '`':
                    line = line[1:]
                # check for ` character at the end of the line
                if line[-1] == '`':
                    line = line[:-1]
            modified_lines.append(line)

    return "\n".join(modified_lines)

def generate_body(cfg: dict)-> None:
    # read CHATGPT_TOKEN from os
    CHATGPT_SESSION_TOKEN = os.getenv("CHATGPT_TOKEN")
    # print(CHATGPT_SESSION_TOKEN)
    session_token = CHATGPT_SESSION_TOKEN  # `__Secure-next-auth.session-token` cookie from https://chat.openai.com/chat
    api = ChatGPT(session_token)  # auth with session token
    output_file = cfg['outputFile']
    sections = cfg['sections']
    with open(output_file, 'a') as f:
        for section in sections:
            # check if str or dict
            if isinstance(section, str):
                resp = api.send_message(section)
                clean_message = use_programming_language(cfg, resp['message'])
            else:
                # assume dict
                # type and src from dict
                input_type = section['type']
                src = section['src']
                if input_type == 'file':
                    # read src from file
                    with open(src, 'r') as src_file:
                        src_text = src_file.read()
                    resp = api.send_message(src_text)
                    clean_message = resp['message']
                    print(clean_message)
                    # write original text
                    programming_language = cfg['programmingLanguage']
                    f.write(f"```
{% endraw %}
{programming_language} \n {src_text} \n
{% raw %}
 ```\n")
                    f.write('\n')
            f.write(clean_message)
            f.write('\n')

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--file', type=str, default='posts/chatgpt_blog_generation.yml')
    args = parser.parse_args()
    # valid files exist
    # argparse for file eventually
    with open(args.file, 'r') as f:
        cfg = load(f, Loader=Loader)
    # if file exists skip
    # if not run these functions
    if os.path.exists(cfg['outputFile']):
        print('file exists')
        exit(0)
    # if genImage is true, then makeImage
    if cfg['imageArgs']:
        generate_image(cfg)
        pass

    generate_frontmatter(cfg)
    generate_body(cfg)

This is a Python script that uses the ChatGPT API, the pyChatGPT library, and the stability_sdk to generate text and images. The script takes in a configuration file in YAML format, which specifies the input text, the output file, and various settings such as the programming language to use for code blocks and the prompt for the image generation.
It uses the ChatGPT API to generate text in response to prompts specified in the configuration file. The script also uses the stability_sdk to generate images based on the prompt specified in the configuration file. The script also uses the Python Imaging Library (PIL) to save the generated images to a directory. Additionally, it processes the generated text, using regular expressions to replace plain backticks with code blocks in the specified programming language.

There are several ways to monetize a blog, including the following:

Advertising: You can place ads on your blog and earn money through pay-per-click or pay-per-impression programs such as Google AdSense.
Affiliate marketing: You can earn money by promoting other people's products or services and receiving a commission for every sale made through your unique affiliate link.
Sponsored posts: You can write posts sponsored by a brand or company and receive payment for the promotion.
Selling products or services: You can sell your own products or services, such as e-books, courses, or consulting services, directly to your readers.
Crowdfunding: You can use platforms like Patreon or Kickstarter to raise money from your readers to support your blog or other projects.
Subscription models: You can offer exclusive content, access to a community or other perks for a recurring fee.

It's important to note that monetizing your blog requires consistent, high-quality content, and a sizable audience. It may take some time and effort to grow your blog and monetize it successfully.

Using AI to generate images can be beneficial in a variety of ways. Some reasons include:

Automation: AI-generated images can be produced quickly and at scale, reducing the time and resources required to create images manually.
Variation: AI-generated images can be easily customized to produce a wide range of variations, allowing for greater flexibility in image design.
Consistency: AI-generated images can be designed to maintain a consistent style or theme, making it easier to maintain a cohesive visual identity across a brand or product.
Cost-effectiveness: AI-generated images can be less expensive than hiring a professional photographer or graphic designer, making them more accessible to businesses with limited budgets.
Creative possibilities: AI-generated images can be used in creative ways, such as creating surreal or abstract imagery that would be difficult to achieve through traditional means.
Personalization: AI-generated images can be used to personalize content for targeted audience, resulting in higher engagement and conversion rates.
Efficiency: AI-generated images can be used to generate high-quality images for use in various applications, such as product advertising, web design, and mobile apps.

Using AI to generate images can be beneficial in a variety of ways. Some reasons include:

Automation: AI-generated images can be produced quickly and at scale, reducing the time and resources required to create images manually.
Variation: AI-generated images can be easily customized to produce a wide range of variations, allowing for greater flexibility in image design.
Consistency: AI-generated images can be designed to maintain a consistent style or theme, making it easier to maintain a cohesive visual identity across a brand or product.
Cost-effectiveness: AI-generated images can be less expensive than hiring a professional photographer or graphic designer, making them more accessible to businesses with limited budgets.
Creative possibilities: AI-generated images can be used in creative ways, such as creating surreal or abstract imagery that would be difficult to achieve through traditional means.
Personalization: AI-generated images can be used to personalize content for targeted audience, resulting in higher engagement and conversion rates.
Efficiency: AI-generated images can be used to generate high-quality images for use in various applications, such as product advertising, web design, and mobile apps.

AI has the potential to change blog writing in the future by making the process more efficient and personalized. For example, AI-powered writing assistants can help writers generate content faster by suggesting topics, outlining structure, and providing research and data to support their ideas. Additionally, AI-powered content generation can personalize blogs by analyzing audience data and tailoring the content to their interests and preferences. It can also generate images, infographics and videos to supplement the written content. However, it's important to note that AI-generated content should be carefully evaluated for accuracy and bias, and should be used in conjunction with human editing and oversight to ensure high-quality and ethical results.

Introduction to LINQ

David Li — Wed, 13 Sep 2023 00:00:00 +0000

LINQ (Language Integrated Query) is a set of language extensions for the Microsoft .NET Framework that allows developers to perform queries on data sources using a syntax similar to SQL. It supports querying data in a variety of formats, including arrays, lists, databases, and XML documents. LINQ was introduced in the .NET Framework version 3.5 and is available in C# and Visual Basic. LINQ makes it easier to write data access code and provides a consistent way to query and manipulate data, regardless of its source.

LINQ queries are composed of a series of operators that can be chained together to filter, transform, and shape data.

The "From" operator specifies the data source for the query. This can be an array, list, or any other collection that implements the IEnumerable interface.
The "Where" operator is used to filter the data based on a certain condition. It returns only the elements that match the specified condition.
The "Select" operator is used to project the data into a new form. It allows you to select only the properties or fields that you are interested in and create a new collection of the results.
The "OrderBy" operator is used to sort the data based on a certain property or field. It returns the data in ascending order by default, but can also be used with the "OrderByDescending" operator to sort in descending order.

Here is an example of a LINQ query that uses these operators:

`var query = from student in students
 where student.Age > 18
 orderby student.LastName, student.FirstName
 select new {student.FirstName, student.LastName};`

This query selects all students who are older than 18, sorts them by last name, first name, and select the firstname and lastname of the students.

LINQ can be used to query data from a List by using the "From" operator to specify the data source and chaining other operators together to filter, transform, and shape the data.

Here is an example of how to use LINQ to query data from a List of integers:

`List<int> numbers = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };

// Use LINQ to query the data
var evenNumbers = from number in numbers
 where number % 2 == 0
 select number;`

In this example, the "From" operator specifies that the data source is the "numbers" List, the "Where" operator filters the data to only include even numbers and "Select" operator selects the even numbers.

You can also use the method syntax

`var evenNumbers = numbers.Where(n => n % 2 == 0).Select(n => n);`

This query will give you the same result as the previous one, but this is the method syntax of LINQ.

You can also use other operators like OrderBy, OrderByDescending, Take, Skip, etc. to shape your data and make it more readable.

`var evenNumbers = numbers.Where(n => n % 2 == 0)
 .OrderBy(n => n)
 .Take(5);`

This query will select the first 5 even numbers from the list and order them by ascending order.

It is important to note that LINQ queries do not execute until they are enumerated, such as by using a foreach loop to iterate through the results.

LINQ can be used to query data from a Dictionary by using the "From" operator to specify the data source and chaining other operators together to filter, transform, and shape the data.

Here is an example of how to use LINQ to query data from a Dictionary:

`Dictionary<string, string> colors = new Dictionary<string, string>
{
 {"red", "#ff0000"},
 {"green", "#00ff00"},
 {"blue", "#0000ff"}
};

// Use LINQ to query the data
var hexColors = from color in colors
 where color.Key == "red" || color.Key == "green"
 select color.Value;`

In this example, the "From" operator specifies that the data source is the "colors" Dictionary, the "Where" operator filters the data to only include the colors which are red and green, and "Select" operator selects the hex value of the colors.

You can also use the method syntax

`var hexColors = colors.Where(c => c.Key == "red" || c.Key == "green")
 .Select(c => c.Value);`

This query will give you the same result as the previous one, but this is the method syntax of LINQ.

You can also use other operators like OrderBy, OrderByDescending, Take, Skip, etc. to shape your data and make it more readable.

It is important to note that when working with a Dictionary, the order of elements is not guaranteed, so if you need to order the elements you need to use the ToList() or ToArray() method before applying OrderBy or OrderByDescending

`var hexColors = colors.Where(c => c.Key == "red" || c.Key == "green")
 .ToList()
 .OrderBy(c => c.Key)
 .Select(c => c.Value);`

This query will select the hex values of red and green colors and order them by their key.

It is important to note that LINQ queries do not execute until they are enumerated, such as by using a foreach loop to iterate through the results.

LINQ can be used to query data from a SQL database by using the LINQ to SQL or the Entity Framework, which are both built on top of LINQ and provide a way to interact with the database using LINQ syntax.

To use LINQ to SQL, you need to create a mapping between the relational database and the object-oriented model. This is done by creating a DataContext class that defines the tables, columns, and relationships in the database. Once the mapping is done you can use LINQ syntax to query the data.

Here is an example of how to use LINQ to SQL to query data from a SQL database:

`using (var db = new DataContext())
{
 var query = from customer in db.Customers
 where customer.Country == "USA"
 select customer;

 foreach (var customer in query)
 {
 Console.WriteLine(customer.FirstName + " " + customer.LastName);
 }
}`

In this example, the "From" operator specifies the data source is the Customers table, the "Where" operator filters the data to only include customers from the USA, and "Select" operator selects the customer.

You can also use the method syntax

`using (var db = new DataContext())
{
 var query = db.Customers.Where(c => c.Country == "USA")
 .Select(c => new {c.FirstName, c.LastName});

 foreach (var customer in query)
 {
 Console.WriteLine(customer.FirstName + " " + customer.LastName);
 }
}`

This query will give you the same result as the previous one, but this is the method syntax of LINQ.

You can also use other operators like OrderBy, OrderByDescending, Take, Skip, etc. to shape your data and make it more readable.

With entities Framework it's similar, but you'll use the "DbContext" class instead of the "DataContext" and you'll need to create a model of the database using EF Core or EF 6.

It is important to note that LINQ queries against a SQL database are executed when the query is enumerated, such as by using a foreach loop to iterate through the results, or by calling the ToList(), ToArray() or Count() methods.

Also remember that it is a good practice to use the using statement to dispose of the DataContext/DbContext and the opened connection to avoid any memory leaks.

LINQ provides a number of advanced operators that can be used to perform more complex queries.

The "GroupBy" operator is used to group elements in a collection based on a certain key. It returns a collection of groups, where each group contains a key and a collection of elements that share that key.

`var query = from student in students
 group student by student.Department into g
 select new {Department = g.Key, Students = g};`

This query groups the students by their department, and returns a collection of groups where each group contains the department name and a collection of students that belong to that department.

The "Join" operator is used to join two collections based on a common key. It returns a collection of elements that have matching keys in both collections.

`var query = from student in students
 join department in departments on student.Department equals department.DepartmentName
 select new {student.FirstName, student.LastName, department.DepartmentName};`

This query joins the students and departments collections by the department name, and returns a collection of elements that contains the student's first name, last name, and department name.

The "Aggregate" operator is used to apply a function to the elements of a collection and return a single value. It can be used to perform calculations such as sum, average, or count.

`var query = students.Where(s => s.Department == "Computer Science")
 .Select(s => s.Grade)
 .Average();`

This query filters the students collection to only include students from the Computer Science department, selects their grades, and calculates the average grade.

It is important to note that these operators can be combined with other LINQ operators, like "Where", "Select", "OrderBy" etc. to create more complex and powerful queries. Also, these operators are executed after the query is enumerated, such as by using a foreach loop to iterate through the results, or by calling the ToList(), ToArray() or Count() methods.